Problem Description:¶
India is seeing an explosion of new competitors in the Broadband space. 'India Broadband' is a company that is now seeing a lot of customer churn due to customer dissatisfaction because of broadband outages.The company has now curated a dataset, where it tracks several variables that it believes impact the outage_duration. They have tracked three different outage durations, 0 for no outage, 1 for short outages that last anywhere between a few minutes and a maximum of 2 hours, and 2 for long outages that can last from 2 hours to sometimes even a couple of days.You will now have to use these metrics that the company has tracked to create a machine learning model that will be able to predict the outage_duration so that the company can better handle outages and improve customer satisfaction and therefore reduce customer churn.¶
Exploratory Data Analysis¶
Importing the required libararies¶
In [1]:
import pandas as pd
import numpy as np
from sklearn import preprocessing
from sklearn.naive_bayes import GaussianNB
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score,classification_report
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import OneHotEncoder
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
import warnings
warnings.filterwarnings('ignore')
from sklearn.naive_bayes import GaussianNB
from sklearn import tree
from sklearn import svm
from sklearn.ensemble import AdaBoostClassifier
#!pip install plotly
import plotly.offline as pyo
import plotly.graph_objs as go
from scipy.stats.mstats import mode
from sklearn.preprocessing import LabelEncoder
%matplotlib inline
from matplotlib.pylab import rcParams
rcParams['figure.figsize'] = 12, 4
Importing Data sets¶
In [2]:
train_data = pd.read_csv('train_data.csv')
test_data = pd.read_csv('test_data.csv')
server_data = pd.read_csv('server_data.csv')
report_data = pd.read_csv('report_data.csv')
broadband_data = pd.read_csv('broadband_data.csv')
outage_data = pd.read_csv('outage_data.csv')
Checking the shape of all Data sets¶
In [3]:
print('The shape of broadband_data is: {}\n'.format(broadband_data.shape))
print('The shape of outage_data is: {}\n'.format(outage_data.shape))
print('The shape of report_data is: {}\n'.format(report_data.shape))
print('The shape of server_data is: {}\n'.format(server_data.shape))
print('The shape of train_data is: {}'.format(train_data.shape))
print('The shape of test_data is: {}'.format(test_data.shape))
Checking the unique columes in the Data sets¶
In [4]:
print(outage_data.nunique())
print('\n')
print(train_data.nunique())
print('\n')
print(broadband_data.nunique())
print('\n')
print(server_data.nunique())
print('\n')
print(report_data.nunique())
print('\n')
print(test_data.nunique())
print('\n')
In [5]:
## Checking the head of all Data sets
print(train_data.head())
print('\n')
print(server_data.head())
print('\n')
print(report_data.head())
print('\n')
print(broadband_data.head())
print('\n')
print(outage_data.head())
print('\n')
print(test_data.head())
print('\n')
Checking the outage duration¶
In [6]:
## outage duration in train_data
train_data['outage_duration'].value_counts()
Out[6]:
In [7]:
## plot for ouatge duration
import seaborn as sns
sns.set_style("whitegrid")
import matplotlib.pyplot as plt
plt.figure(figsize = (14,6))
sns.countplot(train_data['outage_duration'])
plt.tight_layout()
plt.show()
In [8]:
val=list(train_data['outage_duration'].value_counts())
for i in range(len(val)):
print(train_data['outage_duration'].value_counts().index[i],round(val[i]/sum(val)*100),'%')
As there are three different Outage durations¶
- outage durations,
0for no outage - outage durations,
1for short outages that last anywhere between a few minutes and a maximum of 2 hours - outage durations,
2for long outages that can last from 2 hours to sometimes even a couple of days.
Plot with ID and Outage Durations.¶
In [9]:
data = [go.Scatter(x=train_data.loc[train_data.outage_duration==0,'area_code'],
y=train_data.loc[train_data.outage_duration==0,'id'],
mode="markers",text=train_data["outage_duration"]),
go.Scatter(x=train_data.loc[train_data.outage_duration==1,'area_code'],
y=train_data.loc[train_data.outage_duration==1,'id'],
mode="markers",text=train_data["outage_duration"]),
go.Scatter(x=train_data.loc[train_data.outage_duration==2,'area_code'],
y=train_data.loc[train_data.outage_duration==2,'id'],
mode="markers",text=train_data["outage_duration"])]
layout=go.Layout(title="ID vs Area Code",xaxis={"title":"area_code"},yaxis=dict(title="ID"),hovermode="closest")
fig=go.Figure(data,layout)
fig.show()
Preprocessing¶
Checking all the inputs from data sets¶
Broadband_data¶
In [10]:
broadband_data.head()
Out[10]:
In [11]:
broadband_data.shape
Out[11]:
In [12]:
broadband_data['broadband_type'].value_counts()
Out[12]:
In [13]:
##Percentage of Broadband types through Value counts in the Data
val=list(broadband_data['broadband_type'].value_counts())
for i in range(len(val)):
print(broadband_data['broadband_type'].value_counts().index[i],round(val[i]/sum(val)*100),'%')
In [14]:
#count plot for broadband type
plt.figure(figsize = (14,6))
sns.countplot(broadband_data['broadband_type'])
plt.tight_layout()
plt.show()
outage_data¶
In [15]:
outage_data.head()
Out[15]:
In [16]:
outage_data.shape
Out[16]:
In [17]:
outage_data['outage_type'].value_counts()
Out[17]:
In [18]:
#count plot for Outage Duration
plt.figure(figsize = (14,6))
sns.countplot(outage_data['outage_type'])
plt.tight_layout()
plt.show()
In [19]:
## outage_type in percentage
val=list(outage_data['outage_type'].value_counts())
for i in range(len(val)):
print(outage_data['outage_type'].value_counts().index[i],round(val[i]/sum(val)*100),'%')
report_data¶
In [20]:
report_data.head()
Out[20]:
In [21]:
report_data.shape
Out[21]:
In [22]:
report_data.nunique()
Out[22]:
In [23]:
log_report_ = report_data['log_report_type'].value_counts()
In [24]:
log_report_.head(10)
Out[24]:
In [25]:
#count plot for Log Report Type
plt.figure(figsize = (30,8))
sns.countplot(report_data['log_report_type'].value_counts())
plt.tight_layout()
plt.show()
server_data¶
In [26]:
server_data.shape
Out[26]:
In [27]:
server_data.head()
Out[27]:
In [28]:
server_data.nunique().value_counts()
Out[28]:
In [29]:
server_count = server_data['transit_server_type'].value_counts()
server_count
Out[29]:
In [30]:
#plotting with ID vs transit server type
# x and y given as array_like objects
import plotly.express as px
fig = px.scatter(server_data['id'], server_data['transit_server_type'])
fig.show()
In [31]:
server_count.head()
Out[31]:
In [32]:
# plotting server typr count
#count plot for transit server type
plt.figure(figsize = (32,12))
sns.countplot(server_data['transit_server_type'])
plt.tight_layout()
plt.show()
Merging all the data sets with train_data¶
Merging broadband_data with train_data¶
In [33]:
train_broadband_merge = pd.merge(train_data,broadband_data,left_on = ['id'],
right_on = ['id'],how='left')
In [34]:
train_broadband_merge.head()
Out[34]:
In [35]:
train_broadband_merge.shape
Out[35]:
In [36]:
##ploting the data sets to check the inputs with train data
broad_band=train_broadband_merge.broadband_type.value_counts()
broad_band.head(10).plot(kind="bar",figsize=(10,10))
broad_band.head(10)
Out[36]:
Merging server_data with train_data¶
In [37]:
train_server_merge = pd.merge(train_data,server_data,left_on = ['id'],
right_on = ['id'],how='left')
In [38]:
train_server_merge.head()
Out[38]:
In [39]:
train_server_merge.shape
Out[39]:
In [40]:
# plotting server type and outage type
server_outage=train_server_merge.transit_server_type.value_counts()
server_outage.head(10)
server_outage.head(10).plot(kind="bar",figsize=(10,10))
Out[40]:
Merging outage_data with train_data¶
In [41]:
train_outage_merge = pd.merge(train_data,outage_data,left_on = ['id'],
right_on = ['id'],how='left')
In [42]:
train_outage_merge.head()
Out[42]:
In [43]:
train_outage_merge.shape
Out[43]:
In [44]:
# plotting area type and outage duration
outage_duration=train_outage_merge.area_code.value_counts()
outage_duration.head(10)
outage_duration.head(10).plot(kind="bar",figsize=(10,10))
Out[44]:
Merging report_data with train_data¶
In [45]:
train_report_merge = pd.merge(train_data,report_data,left_on = ['id'],
right_on = ['id'],how='left')
In [46]:
train_report_merge.head()
Out[46]:
In [47]:
train_report_merge.shape
Out[47]:
In [48]:
# plotting log_report_type and outage duration
report_outage=train_report_merge.log_report_type.value_counts()
report_outage.head(10)
report_outage.head(10).plot(kind="bar",figsize=(10,10))
Out[48]:
Feature engineering using train and test_data¶
In [49]:
## Adding a column in train_data
train_data['train'] = 'yes'
test_data['train'] = 'no'
In [50]:
# Checking the head of train and test_data
print(train_data.head())
print('\n')
print(test_data.head())
print('\n')
merging the train_data and test_data¶
In [51]:
temp_data = pd.concat([train_data,test_data], ignore_index=True)
In [52]:
# Checking the train and test_data shape after merge
temp_data.shape
Out[52]:
In [53]:
temp_data.head()
Out[53]:
In [54]:
temp_data.tail()
Out[54]:
In [55]:
## arranging the columns
data_columns=["id","area_code","train","outage_duration"]
data_columns
Out[55]:
In [56]:
data = temp_data[data_columns]
In [57]:
data.head()
Out[57]:
Merging trian_test_data with server_data¶
In [58]:
server_data=server_data.merge(data,on='id',how='left')
In [59]:
## Checking the top rows of the server data after merging with train and test data
server_data.head()
Out[59]:
In [60]:
server_data_temp=pd.DataFrame(server_data['transit_server_type'].value_counts())
In [61]:
## checking the top most transit_server_type
server_data_temp.head()
Out[61]:
In [62]:
server_data_temp.nunique()
Out[62]:
In [63]:
## checking how percentage of transit_server_type involved in the train_data
server_data_temp['Perc_Train'] = server_data.pivot_table(values='train',index='transit_server_type',aggfunc=lambda x: sum(x=='yes')/float(len(x)))
server_data_temp.head()
Out[63]:
In [64]:
server_data_temp
Out[64]:
In [65]:
server_data_temp['Mode_outage_duration'] = server_data.loc[server_data['train']=='yes'].pivot_table(values='outage_duration',index='transit_server_type', aggfunc=lambda x: mode(x).mode[0])
In [66]:
server_data_temp.iloc[-10:]
Out[66]:
In [67]:
server_data_temp.nunique()
Out[67]:
In [68]:
server_data_temp['preprocess'] = server_data_temp.index.values
In [69]:
bottomlimit = 30
server_data_temp['preprocess'].iloc[bottomlimit:] = server_data_temp['Mode_outage_duration'].iloc[bottomlimit:].apply(lambda x: 'Remove' if pd.isnull(x) else 'transit_server_other_%d'%int(x))
In [70]:
server_data_temp['preprocess'].iloc[33:]
Out[70]:
In [71]:
print (server_data_temp['preprocess'].value_counts())
print (server_data_temp)
In [72]:
server_data = server_data.merge(server_data_temp[['preprocess']], left_on='transit_server_type',right_index=True)
print (server_data.head())
server_data['preprocess'].value_counts()
Out[72]:
In [73]:
server_data_merge = server_data.pivot_table(values='transit_server_type',index='id',columns='preprocess',aggfunc=lambda x: len(x), fill_value=0)
server_data_merge.shape
Out[73]:
In [74]:
server_data_merge.columns
Out[74]:
In [75]:
server_data_merge.head()
Out[75]:
In [76]:
server_data.shape
Out[76]:
In [77]:
server_data_train=data.merge(server_data_merge,left_on='id',right_index=True)
server_data_train.head()
Out[77]:
In [78]:
server_data_train.shape
Out[78]:
In [79]:
server_temp=server_data.copy()
In [80]:
server_temp=server_temp.drop(['preprocess'],axis=1)
In [81]:
server_temp.shape
Out[81]:
In [82]:
server_temp_merge=server_temp.merge(server_data_merge,left_on='id',right_index=True)
server_temp_merge.shape
Out[82]:
In [83]:
server_temp_merge.duplicated('id').sum()
Out[83]:
In [84]:
server_temp_merge.loc[server_temp_merge.duplicated(subset='id',keep=False),:]
Out[84]:
In [85]:
server_data_dup=server_data.merge(server_data_merge,left_on='id',right_index=True)
server_data_dup.head()
Out[85]:
In [86]:
server_data_dup.shape
Out[86]:
In [87]:
server_data_dup.loc[server_data_dup.duplicated(subset='id',keep=False),:]
Out[87]:
Merging train and test _data with report_data¶
In [88]:
data.head()
Out[88]:
In [89]:
report_data.head()
Out[89]:
In [90]:
report_data.shape
Out[90]:
In [91]:
data.shape
Out[91]:
In [92]:
report_data=report_data.merge(data,on='id',how='left')
report_data.head()
Out[92]:
In [93]:
report_data_temp=pd.DataFrame(report_data['log_report_type'].value_counts())
report_data_temp.head()
Out[93]:
In [94]:
report_data_temp['Perc_Train'] = report_data.pivot_table(values='train',index='log_report_type',aggfunc=lambda x: sum(x=='yes')/float(len(x)))
report_data_temp.head()
Out[94]:
In [95]:
report_data_temp['Mode_outage_duration']=report_data.loc[report_data['train']=='yes'].pivot_table(values='outage_duration',index='log_report_type',aggfunc=lambda x: mode(x).mode[0])
In [96]:
len(report_data_temp)
Out[96]:
In [97]:
report_data_temp['preprocess']=report_data_temp.index.values
In [98]:
report_data_temp['preprocess'].loc[report_data_temp['Perc_Train']==1] = np.nan
In [99]:
report_data_temp[-20:]
Out[99]:
In [100]:
report_data_temp[:5]
Out[100]:
In [101]:
limit=140
report_data_temp['preprocess'].iloc[limit:]=report_data_temp['Mode_outage_duration'].iloc[limit:].apply(lambda x: 'Remove' if pd.isnull(x) else 'log_report_other%d'%int(x))
print (report_data_temp['preprocess'].value_counts())
report_data_temp
Out[101]:
In [102]:
report_data=report_data.merge(report_data_temp[['preprocess']],left_on='log_report_type',right_index=True)
print(report_data.head())
report_data['preprocess'].value_counts()
Out[102]:
In [103]:
report_data_merge = report_data.pivot_table(values='volume',index='id',columns='preprocess',aggfunc=np.sum, fill_value=0)
report_data_merge.shape
Out[103]:
In [104]:
report_data_merge.sum().sum()
Out[104]:
In [105]:
report_data_merge.head(-20)
Out[105]:
In [106]:
server_report_train=server_data_train.merge(report_data_merge,left_on='id',right_index=True)
print(server_report_train.shape)
server_report_train.head()
Out[106]:
Merging train and test _data with outage_data¶
In [107]:
outage_data.head()
Out[107]:
In [108]:
data.head()
Out[108]:
In [109]:
outage_data['outage_type'].value_counts()
Out[109]:
In [110]:
outage_data=outage_data.merge(data,on='id',how='left')
outage_data.head()
Out[110]:
In [111]:
outage_data.shape
Out[111]:
In [112]:
outage_data_temp = pd.DataFrame(outage_data['outage_type'].value_counts())
outage_data_temp.head()
Out[112]:
In [113]:
outage_data_temp['Perc_Train']=outage_data.pivot_table(values='train',index='outage_type',aggfunc=lambda x: sum(x=='yes')/float(len(x)))
outage_data_temp.head()
Out[113]:
In [114]:
outage_data_temp['Mode_outage_duration']=outage_data.loc[outage_data['train']=='yes'].pivot_table(values='outage_duration',index='outage_type',aggfunc=lambda x: mode(x).mode[0])
outage_data_temp
Out[114]:
In [115]:
outage_data.loc[server_data['train']=='yes'].pivot_table(values='outage_duration',index='outage_type', aggfunc=lambda x: mode(x))
Out[115]:
In [116]:
outage_data_merge = outage_data.pivot_table(values='train',index='id',columns='outage_type',aggfunc=lambda x: len(x), fill_value=0)
outage_data_merge.head()
Out[116]:
Merging train and test_data with broadband_data¶
In [117]:
broadband_data.head()
Out[117]:
In [118]:
broadband_data.shape
Out[118]:
In [119]:
broadband_data['broadband_type'].value_counts()
Out[119]:
In [120]:
broadband_data = broadband_data.merge(data, on='id',how='left')
broadband_data.head()
Out[120]:
In [121]:
broadband_data.shape
Out[121]:
In [122]:
broadband_data_temp = pd.DataFrame(broadband_data['broadband_type'].value_counts())
broadband_data_temp.head()
Out[122]:
In [123]:
broadband_data_temp['PercT_rain'] = broadband_data.pivot_table(values='train',index='broadband_type',aggfunc=lambda x: sum(x=='yes')/float(len(x)))
broadband_data_temp
Out[123]:
In [124]:
broadband_data_temp['Mode_outage_duration'] = broadband_data.loc[broadband_data['train']=='yes'].pivot_table(values='outage_duration',index='broadband_type', aggfunc=lambda x: mode(x).mode[0])
broadband_data_temp
Out[124]:
In [125]:
broadband_data.loc[broadband_data['broadband_type']=='broadband_type_5']
Out[125]:
In [126]:
broadband_data_merge = broadband_data.pivot_table(values='train',index='id',columns='broadband_type',aggfunc=lambda x: len(x), fill_value=0)
broadband_data_merge.head()
Out[126]:
In [127]:
server_report_broadband_train=server_report_train.merge(broadband_data_merge,left_on='id',right_index=True)
print(server_report_broadband_train.shape)
server_report_broadband_train.head()
Out[127]:
In [128]:
server_report_broadband_outage_train=server_report_broadband_train.merge(outage_data_merge,left_on='id',right_index=True)
server_report_broadband_outage_train.head()
Out[128]:
In [129]:
server_report_broadband_outage_train.dtypes
Out[129]:
In [130]:
server_report_broadband_outage_train.shape
Out[130]:
In [131]:
broadband_data['broadband_type'].value_counts()
Out[131]:
In [132]:
report_data['log_report_type'].value_counts()
Out[132]:
In [133]:
report_data.head()
Out[133]:
In [134]:
train_outage_2=train_data.loc[train_data.outage_duration==2]
In [135]:
server_report_broadband_outage_train.shape
Out[135]:
In [136]:
broadband_outage2=broadband_data.loc[broadband_data.outage_duration==2]
In [137]:
broadband_outage2.shape
Out[137]:
In [138]:
broadband_outage2.head()
Out[138]:
Visualization¶
The below plot shows the area code with long outage durations¶
In [139]:
import plotly
import plotly.offline as pyoff
# import plotly.figure_factory as ff
from plotly.offline import init_notebook_mode, iplot, plot
import plotly.graph_objs as go
%matplotlib notebook
In [140]:
temp = train_outage_2.area_code.value_counts()
In [141]:
data = [go.Bar(
x=temp.index,
y=temp)]
iplot(data)
In [142]:
broadband_temp= broadband_outage2.broadband_type.value_counts()
In [143]:
broadband_temp
Out[143]:
In [144]:
data = [go.Bar(
x=broadband_temp.index,
y=np.round(broadband_temp.astype(float)/broadband_temp.values.sum(),2),
text = np.round(broadband_temp.astype(float)/broadband_temp.values.sum(),2),
textposition = 'auto')]
iplot(data)
Above plot shows that broadband type 8 i.e. ADSL-1 is more susceptable to long outage duration¶
Based on domain knowledge we know that the types of broadband connects can be classified into Digital subscriber line(DSL), Cable modem, Fibre and Broadband over powerlines (BPL).¶
In [145]:
broadband_data['last_digit']=broadband_data['broadband_type'].apply(lambda x: int(x.split("_")[2]))
In [146]:
broadband_data.head()
Out[146]:
In [147]:
broadband_data['classification']=broadband_data['last_digit'].apply(lambda x: 'DSL' if x==8 else ('DSL' if x==2 else('DSL' if x==6 else('Cable' if x==7 else('BPL' if x==9 else 'Fibre')))))
In [148]:
broadband_data.head()
Out[148]:
In [149]:
broadband_classified=broadband_data.loc[broadband_data.outage_duration==2].classification.value_counts()
In [150]:
data = [go.Bar(
x=broadband_classified.index,
y=np.round(broadband_classified.astype(float)/broadband_classified.values.sum(),2),
text = np.round(broadband_classified.astype(float)/broadband_classified.values.sum(),2),
textposition = 'auto')]
iplot(data)
We can see that DSL type of broadband connection is having more outage duration(93%) than others in above plot.¶
Below plot shows the broadband connection preference.¶
In [151]:
broadband_alloutage=broadband_data.broadband_type.value_counts()
In [152]:
data = [go.Bar(
x=broadband_alloutage.index,
y=np.round(broadband_alloutage.astype(float)/broadband_alloutage.values.sum(),2),
text = np.round(broadband_alloutage.astype(float)/broadband_alloutage.values.sum(),2),
textposition = 'auto')]
iplot(data)
In [153]:
broadband_alloutage_classified=broadband_data.classification.value_counts()
In [154]:
data = [go.Bar(
x=broadband_alloutage_classified.index,
y=np.round(broadband_alloutage_classified.astype(float)/broadband_alloutage_classified.values.sum(),2),
text = np.round(broadband_alloutage_classified.astype(float)/broadband_alloutage_classified.values.sum(),2),
textposition = 'auto')]
iplot(data)
In [155]:
outage_data.head()
Out[155]:
Count plot for Outage type¶
In [156]:
outage_data_plot=outage_data.dropna()
In [157]:
outage_data_plot.head()
Out[157]:
Below plot shows how each outage type has contributed to outage duration, also we can infer that outage_type 1 and 2 are contributing more to long outage duration.¶
In [158]:
plt.figure(figsize=(10,5))
sns.countplot(x="outage_type", hue="outage_duration", data=outage_data_plot)
Out[158]:
In [159]:
report_data.head()
Out[159]:
In [160]:
report_volume=report_data.groupby('log_report_type').volume.sum()
In [161]:
report_largest=report_data.groupby('log_report_type').volume.sum().nlargest(30)
Below is the plot for top 30 log_report_type based on volume¶
In [162]:
data = [go.Bar(
x=report_largest.index,
y=report_largest,
text = report_largest,
textposition = 'auto')]
iplot(data)
In [163]:
server_data.transit_server_type.value_counts()
Out[163]:
In [164]:
server_plot=server_data.preprocess.value_counts()
Below plot shows transit_server_type count plot with only 34 features. Rest are combined into 3 new features.¶
In [165]:
data = [go.Bar(
x=server_plot.index,
y=server_plot,
text = server_plot,
textposition = 'auto')]
iplot(data)
In [166]:
server_report_broadband_outage_train.shape
Out[166]:
In [167]:
server_report_broadband_outage_train.head()
Out[167]:
In [168]:
[x for x in server_report_broadband_outage_train.columns if 'Remove' in x]
Out[168]:
In [169]:
server_report_broadband_outage_train.drop(['Remove_x','Remove_y'],axis=1,inplace=True)
In [170]:
server_report_broadband_outage_train.head()
Out[170]:
In [171]:
server_report_broadband_outage_train.shape
Out[171]:
In [172]:
data_new=server_report_broadband_outage_train.copy()
In [173]:
x=data_new.pop('outage_duration')
In [174]:
data_new['outage_duration']=x
data_new.head()
Out[174]:
train_new is the new train data after EDA¶
We can get better insights on outage duration based on log_report_type and transit_server_type. We have to see the outage duration for the logreport with large volumns and also check trainsit_server counts and relate it to outage duration. This can help us with detecting outage duration better.¶
In [175]:
data_new.shape
Out[175]:
In [176]:
data_new.dtypes
Out[176]:
In [177]:
data_new.head()
Out[177]:
In [178]:
le = LabelEncoder()
data_new['area_code'] = le.fit_transform(data_new['area_code'])
In [179]:
data_new.head()
Out[179]:
In [180]:
train=data_new.loc[data_new.train=='yes']
In [181]:
train.tail(5)
Out[181]:
In [182]:
train=train.drop('train',axis=1)
train.head()
Out[182]:
In [183]:
test=data_new.loc[data_new.train=='no']
In [185]:
test=test.drop('train',axis=1)
test.head()
Out[185]:
Train Test Split¶
In [186]:
X, Y = train.loc[:,train.columns!='outage_duration'], train.loc[:,'outage_duration']
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, stratify =Y,random_state=123)
logistic regression¶
In [187]:
from sklearn.metrics import accuracy_score, classification_report
In [188]:
clf = LogisticRegression()
clf = clf.fit(X_train, Y_train)
traine_pred = clf.predict(X_train)
teste_pred = clf.predict(X_test)
In [189]:
print(classification_report(Y_test,teste_pred))
In [190]:
print(classification_report(Y_train,traine_pred))
Random Forest¶
In [191]:
clf1=RandomForestClassifier()
clf1 = clf1.fit(X_train, Y_train)
traine_pred = clf1.predict(X_train)
teste_pred = clf1.predict(X_test)
In [192]:
print(classification_report(Y_train, traine_pred))
print(classification_report(Y_test, teste_pred))
decision tree¶
In [193]:
clf2 = tree.DecisionTreeClassifier()
clf2 = clf2.fit(X_train, Y_train)
traine_pred_df = clf2.predict(X_train)
teste_pred_df = clf2.predict(X_test)
In [194]:
print(classification_report(Y_train,traine_pred_df))
print('\n')
print('\n')
print(classification_report(Y_test,teste_pred_df))
Grid Search cv¶
In [195]:
param_grid = {"criterion": ['entropy'],
'min_samples_split': [5,10,20],
'max_depth': [2,5,10,15,30],
'max_leaf_nodes': [100,120,135,150], }
In [196]:
dt = tree.DecisionTreeClassifier()
clf_3_cv = GridSearchCV(dt, param_grid, cv = 5,n_jobs=-1)
clf_3_cv.fit(X_train, Y_train)
traine_pred = clf_3_cv.predict(X_train)
teste_pred = clf_3_cv.predict(X_test)
In [197]:
clf_3_cv.best_params_
Out[197]:
In [198]:
print(classification_report(Y_train,traine_pred))
print('\n')
print('\n')
print(classification_report(Y_test,teste_pred))
Navie Bayes¶
In [199]:
clf4 =GaussianNB()
clf4 = clf4.fit(X_train, Y_train)
traine_pred_df = clf4.predict(X_train)
teste_pred_df = clf4.predict(X_test)
In [200]:
print(classification_report(Y_train, traine_pred))
print(classification_report(Y_test, teste_pred))
AdaBoostClassifier¶
In [201]:
clf5=AdaBoostClassifier()
clf5 = clf5.fit(X_train, Y_train)
traine_pred_df = clf5.predict(X_train)
teste_pred_df = clf5.predict(X_test)
In [202]:
print(classification_report(Y_train, traine_pred))
print(classification_report(Y_test, teste_pred))
In [203]:
test=test.drop('outage_duration',axis=1)
In [204]:
test.head()
Out[204]:
In [205]:
test_pred_1=clf_3_cv.predict(test)
In [206]:
test_pred_1
Out[206]:
In [207]:
final_dt=pd.DataFrame({'id':test['id'],'outage_duration':test_pred_1})
In [208]:
final_dt.head()
Out[208]:
In [209]:
final_dt.reset_index()
Out[209]:
In [210]:
final_dt.reset_index(drop=True)
Out[210]:
In [212]:
## grid search cv
final_dt.to_csv("C:/Users/admin/Desktop/Mohammed Final Exam//Submission_1_PHD.csv",index=False)
In [213]:
test_pred_2 = clf4.predict(test)
In [214]:
test_pred_2
Out[214]:
In [215]:
final_dt2=pd.DataFrame({'id':test['id'],'outage_duration':test_pred_2})
In [216]:
# GaussianNB
final_dt2.to_csv("C:/Users/admin/Desktop/Mohammed Final Exam//Submission_2_PHD.csv",index=False)